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SUMMARY 


Background 

The  impact  of  new  systems  on  human  resource  requirements  should  be 
assessed  as  early  as  possible  during  weapon  system  acquisition.  Recent 
studies  have  suggested  that  estimates  of  maintenance,  manpower,  and 
training  requirements  for  Air  Force  systems  can  be  made  in  the  early 
design  stages  by  technicians  with  maintenance  experience  on  similar, 
operational  systems. 

Objective 

The  objective  of  this  effort  was  to  evaluate  and  revise  a 
previously  developed  prototype  users  guide  for  collecting  expert 
estimates  of  maintenance,  manpower,  and  training  data  for  new  or 
proposed  Air  Force  systems. 

Approach 

The  revision  of  the  prototype  users  guide  was  based  on  infor¬ 
mation  from  three  separate  efforts:  a  review  of  recent  expert 
estimate  studies,  an  application  of  the  expert  estimate  method  using 
the  prototype  users  guide,  and  an  independent  evaluation  of  the 
guide  by  experienced  manpower  professionals. 

Results 


A  revised  users  guide  was  produced.  It  contains  information  on 
the  background  of  the  expert  estimate  method  and  its  capabilities, 
limitation,  and  applications.  Specific  areas  covered  include  the 
development  of  the  engineering  description  package  and  the  ques¬ 
tionnaire,  procedures  for  selecting  estimators  and  collecting  the 
estimates,  and  data  reduction  and  analysis  methods. 
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EXPERT  ESTIMATE  METHOD  OF  GENERATING  MAINTENANCE 
AND  MANPOWER  DATA  FOR  PROPOSED  AIR  FORCE  SYSTEMS:  USERS  GUIDE 

INTRODUCTION 


The  Expert  Estimate  Method 

There  are  several  techniques  and  methods  available  for  pre¬ 
dicting  the  human  resource  requirements  for  new  Air  Force  systems. 
Among  these  techniques  are  the  historical  comparison  method,  task- 
analytic  method,  sovereign  factors,  modeling  techniques  and  expert 
estimate  techniques  (Haines  &  Gael,  1963).  The  first  four  methods 
require  vast  amounts  of  data,  complex  machine  processing,  or  the 
construction  of  sophisticated  models  to  produce  the  human  resource 
estimates.  The  expert  estimate  approach  requires  relatively  little 
in  terms  of  external  support  and,  therefore,  represents  a  relatively 
low-cost  method  for  producing  human  resource  estimates.  The  data 
base  for  the  expert  estimate  approach  is  made  up  of  the  years  of 
systems  experience  accumulated  by  Air  Force  technicians.  It  is  this 
data  base  that  the  expert  estimate  technique  can  rapidly  query  with 
small  investments  of  resources. 

The  expert  estimate  technique  consists  of  four  basic  steps. 
First,  an  engineering  description  package  is  compiled  for  the  new  or 
proposed  equipment  or  system  under  study.  This  description  is  based 
on  the  engineering  data  and  specifications  available  during  the 
early  phases  of  system  design.  Second,  a  questionnaire  is  designed 
to  collect  the  specific  human  resource  estimates  desired.  The  third 
step  is  to  collect  the  estimates.  This  includes  the  selection  of 
the  appropriate  kinds  and  quantities  of  technicians  to  serve  as 
expert  estimators  and  the  completion  of  the  data  collection  visits. 
The  fourth  step  is  to  reduce  and  analyze  the  data.  These,  then,  are 
the  basic  steps  involved  in  using  the  expert  estimate  method.  Each 
of  these  steps  will  be  discussed  in  more  detail  in  later  sections  of 
this  guide. 

Capabilities/Limitations  of  the 
Expert  Estimate  Method 

Recent  research  (Sauer  &  Askren,  1978;  Sauer,  Deem,  &  Askren, 
1979;  and  Whalen  &  Askren,  1974)  on  the  expert  estimate  method  has 
been  directed  toward  determining  the  accuracy,  reliability,  and 
generalizability  of  the  method  for  different  types  of  human  resource 
data  and  different  types  of  system  technologies.  The  research  has 
shown  that  the  method  can  produce  accurate  and  reliable  estimates  of 

1.  Maintenance  Task  Times 

2.  Maintenance  Person-Hours 
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3.  Crew  Size 

4.  Skill  Level 

5.  Career  Field/Air  Force  Specialty  Code  (AFSC) 

6.  Task  Difficulty 

7.  Training  Times 

The  method  has  also  been  used  to  collect  estimates  of  support  equip¬ 
ment  requirements  and  estimates  of  the  relative  frequency  of  various 
types  of  maintenance  tasks  which  may  be  required  for  a  new  component 
or  system.  Technicians  participating  in  the  research  efforts  to 
date,  however,  have  not  been  able  to  produce  satisfactory  estimates 
of  these  types  of  data. 

Applications  for  the  Expert  Estimate  Method 

The  expert  estimate  approach  to  predicting  human  resources  for 
new  systems  does  not  depend  on  the  availability  of  prototype  or 
actual  system  equipment.  Estimates  can  be  made  with  only  an  engi¬ 
neering  description  of  the  proposed  system.  For  this  reason,  the 
expert  estimate  approach  is  ideal  for  use  in  the  early  design  stages 
of  new  systems  before  prototypes  are  constructed.  The  impact  of  the 
system  design  on  the  human  resources  can  be  assessed  and,  if  neces¬ 
sary,  the  design  can  be  modified.  Since  system  changes  in  the  early 
design  stages  can  be  effected  more  easily  and  are  less  costly  than 
similar  changes  made  later  in  the  system  development  process,  the 
expert  estimate  method  can  play  a  significant  role  in  the  process  of 
optimizing  system  cost  and  system  performance. 

The  expert  estimate  method  could  also  be  used  to  compare  the 
human  resource  impacts  of  two  or  more  alternative  systems,  designs, 
or  engineering  solutions.  Another  application  could  be  in  projecting 
personnel  costs  and  personnel  needs  for  new  systems.  It  could  also 
be  used  to  assist  career  field  planners  and  training  planners.  The 
method  could  be  used  as  a  means  of  supplementing  the  data  bases  used 
for  more  sophisticated  human  resource  estimation  techniques  or 
estimation  models.  Further  research  is  expected  to  refine  and 
expand  the  capabilities  of  the  method  as  well  as  to  define  more 
clearly  its  most  advantageous  applications  and  limitations. 

The  following  sections  of  the  guide  describe  in  detail  the 
steps  necessary  to  apply  the  expert  estimate  method.  Where  neces¬ 
sary,  examples  are  provided  to  supplement  the  explanations  of  the 
steps. 
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DEVELOPING  THE  ENGINEERING  DESCRIPTION  PACKAGE 


The  first  step  in  preparing  to  use  the  expert  estimate  method 
is  to  develop  the  engineering  description  package.  This  package 
will  be  the  only  source  of  information  on  the  proposed  system  or 
equipment  that  the  technician  will  have.  As  such,  it  is  critical 
that  as  much  pertinent  information  as  possible  be  presented  in  the 
package.  Generally,  the  engineering  description  package  should 
contain  the  following  types  of  information: 

1.  Type  of  System — Identify  the  type  of  system  for  which 
estimates  are  to  be  made,  for  example,  a  strategic  bomber 
system,  a  cruise  missile  system,  a  hydraulic  system,  a 
mobile  ground  radar  system,  or  an  electronic  counter¬ 
measures  system. 

2.  Location  of  Components  Within  the  System — Indicate  where 
individual  components  are  physically  located  within  the 
system.  Also  indicate  the  spatial/physical  relationships 
among  system  components  and  between  system  components  and 
other  systems. 

3.  Access  to  Components — Indicate,  through  detailed  text  or 
graphics,  the  extent  to  which  other  components,  panels,  or 
hatches  must  be  removed  to  gain  access  to  the  system 
components. 

A.  Physical  Description  of  Components — Size  and  Weight — 

Indicate  component  exterior  dimensions,  weight,  shape,  and 
any  unusual  mounting  or  attachment  hardware  associated 
with  the  component. 

5.  Functional  Description  of  Components — Describe  what  each 
component  does,  its  input  data  and  its  output  data. 
Schematic  or  block  diagrams  are  often  helpful  to  describe 
the  functional  relationship  among  components. 

6.  Description  of  Built-In  Test  Capability — Describe  what  can 
be  tested,  when  in  the  operation  of  the  system  the  tests 
can  be  made,  and  how  and  in  what  form  test  results  are 
presented. 

7.  Description  of  Test  Equipment — Describe  the  types  of  test 
equipment  other  than  built-in  test  equipment  that  are 
required  for  the  system.  Include  generic  names  of  test 
equipment  and  brand  names  if  appropriate.  Identify  any 
special  adapters,  special  power  requirements,  environ¬ 
mental  conditions,  or  other  special  support  equipment  or 
facilities . 
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8.  Maintainability — Include  in  the  engineering  description 

package  any  preliminary  estimates  of  repair  or  maintenance 
times  available  during  the  early  design  stages. 

These  kinds  of  system  information  may  be  obtained  from  system 
designers  and  engineers  through  letters,  interviews,  briefings, 
conferences,  or  any  other  means  available  to  transfer  the  engi¬ 
neering  information  to  the  user  of  the  expert  estimate  method. 

It  is  possible  that  many  of  the  details  will  not  be  known  or 
will  nor  have  been  decided  during  the  early  design  stages;  there¬ 
fore,  the  engineering  description  package  should  be  developed  from 
the  data  that  are  avu;lable. 

There  is  no  set  format  for  the  engineering  description  package. 
Experience  has  shown  that  schematics,  block  diagrams,  flow  diagrams, 
drawings,  and  other  graphic  aids  were  very  helpful  to  the  tech¬ 
nicians  who  have  used  the  packages.  The  length  of  the  engineering 
packages  used  in  the  expert  estimate  research  ranged  from  10  to 
15  pages  of  text  and  diagrams.  Most  of  the  technicians  felt  that 
these  lengths  were  about  right  and  some  stated  that  they  could  have 
been  longer.  After  the  description  package  has  been  compiled,  it  is 
advisable  to  have  a  small  group  (5  to  10)  of  potential  technician 
estimators  review  the  package  to  ensure  that  all  technical  and 
engineering  terms  are  clearly  defined  and  understood. 

The  main  cost  of  developing  an  engineering  package  will  be  the 
time  necessary  to  collect,  edit,  and  compile  the  engineering  data  in 
a  single  document.  This  includes  the  person-hours  spent  by  the 
individual  charged  with  preparing  the  engineering  description  pack¬ 
age;  the  engineers  or  designers  who  will  furnish  the  engineering 
data  on  the  proposed  system;  drafting  and  artistic  support;  and 
typing,  editing,  and  reproduction  support.  Travel  expenses  should 
also  be  considered  if  the  data  collector  needs  to  visit  distant 
locations  to  obtain  required  data. 
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DEVELOPING  THE  QUESTIONNAIRE 


The  development  of  the  questionnaire  is  a  process  of  selecting 
the  appropriate  question  formats  for  the  types  of  human  resource  < 

estimates  desired.  Before  discussing  the  specific  question  formats,  j 

it  is  necessary  to  address  the  construction  of  maintenance  scenarios. 

The  maintenance  scenario  serves  as  the  frame  of  reference  for  the  1 

task  time,  crew  size,  skill  level,  task  difficulty  estimates,  and 
derived  person-hour  estimates.  The  scenario  presents  information 
similar  to  what  the  maintenance  technicians  encounter  in  their  daily 
AFM  66-1  recordkeeping  activities.  A  maintenance  scenario  contains 
at  least  four  types  of  information:  (a)  the  identification  of  the 
component  or  subsystem  (e.g..  Work  Unit  Code),  (b)  identification  of 
maintenance  type  (on-  or  off-equipment),  (c)  identification  of  the 
specific  malfunction  (e.g..  How  Malfunction  Code),  and  (d)  identi¬ 
fication  of  the  specific  maintenance  action  taken  (e.g..  Action 
Taken  Code).  The  Work  Unit  Code  manuals  for  similar  operational 
systems  may  be  consulted  to  determine  the  types  of  malfunctions  and 
maintenance  actions  which  may  be  included  for  the  proposed  system. 

Maintenance  scenarios  may  be  specified  for  as  many  system  components, 
maintenance  types,  malfunctions,  or  maintenance  actions  as  necessary 
for  the  particular  application  of  the  expert  estimate  method.  If, 
for  example,  a  system  was  composed  of  10  components  and  information 
was  desired  for  five  on-equipment  and  five  off-equipment  malfunc¬ 
tions  and  maintenance  actions,  it  would  be  necessary  to  specify  100 
maintenance  scenarios  (10  components  *  5  malfunctions  and/or  mainte¬ 
nance  actions  *  2  maintenance  types). 

Maintenance  Task  Time  Estimates 


Maintenance  task  time  estimates  should  be  made  for  each  mainte¬ 
nance  scenario  constructed.  To  help  technicians  estimate  the  normal 
task  time  for  a  given  maintenance  action,  hav  '  technicians  consider 
the  full  range  of  possible  task  times.  The  technicians  should 
consider  three  task  times:  (a)  the  minimum  time  in  which  the  task 
could  be  completed,  (b)  the  normal  time  in  which  the  'task  would  be 
completed,  and  (c)  the  maximum  time  for  task  completion.  The  fol¬ 
lowing  is  an  example  of  a  format  which  has  been  used  successfully  in 
pri'Vious  research  efforts.  The  four  parts  of  the  maintenance 
scenario  are  presented  prior  to  the  actual  task  time  format. 
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Maintenance  Scenario 


Component :  Transmitter  (WUC-ABAOO) 
Type  Maintenance:  On-Equipment 
How  Malfunction  Code:  No  Output 
Action  Taken  Code:  Troubleshoot 


Minimum 

Time  _ 1. 2 _ 

(hours  &  tenths 
of  hours) 


Normal 

Time  _ 1. 8 _ 

(hours  &  tenths 
of  hours) 


Maximum 

Time  _ 2.  5 _ 

(hours  &  tenths 
of  hours) 


Crew  Size  Estimates 

Crew  size  estimates  can  be  made  in  terms  of  the  percentage  of 
times  a  particular  task  will  require  a  crew  of  one,  two,  three,  or 
more  members.  Present  the  possible  crew  sizes  for  a  particular 
maintenance  scenario  and  have  the  technicians  indicate  next  to  each 
crew  size  the  percentage  of  times  that  size  crew  would  be  required 
to  perform  the  maintenance  task.  The  following  is  an  example  of  a 
format  (with  typical  estimates)  for  collecting  crew  size  estimates. 


Maintenance  Scenario 

Component :  Transmitter  (WUC-ABAOO) 
Type  Maintenance :  On-Equipment 
How  Malfunction  Code:  No  Output 
Action  Taken  Code:  Troubleshoot 


Crew  Size 
1 
2 
3 
A 


Percent  of  Time 
85 
10 
J5 
0 


The  estimator  predicts  that  85  percent  of  the  time  that  this  task 
occurs,  one  person  can  perform  the  task;  10  percent  of  the  time  the 
task  occurs,  two  persons  are  needed  to  perform  the  task;  and  5  per¬ 
cent  of  the  time  the  task  occurs,  three  persons  are  needed  to  per¬ 
form  the  task.  The  task  will  never  require  a  crew  of  four  persons 
according  to  the  estimates  presented  here. 


Person-Hour  Values 


In  the  expert  estimate  method,  technicians  do  not  estimate 
person-hours  directly.  When  recording  daily  maintenance  activities, 
technicians  enter  task  time  and  crew  size  on  the  AFTO  form  349 
(Maintenance  Data  Collection  Records).  They  are  not  required  to 
think  in  terms  of  person-hours  nor  are  data  entered  in  terms  of 
person-hours.  For  the  expert  estimate  method,  person-hour  values 
are  derived  from  the  task  time  estimates  and  the  crew  size  esti¬ 
mates.  Specifically,  each  technician's  estimates  of  normal  task 
time  and  of  the  crew  size  most  likely  to  perform  the  task  are  used 
to  calculate  person-hour  values.  For  example,  if  a  technician 
estimates  that  the  normal  task  time  for  a  given  task  is  1.9  hours 
and  that  75  percent  of  the  time  a  two-member  crew  is  required,  the 
person-hour  value  resulting  from  these  estimates  is  3.8  person-hours 
(1.9  hours  x  2  crewmembers).  A  group  mean  person-hour  value  is 
calculated  by  finding  the  mean  of  the  individual  person-hour  values. 
Person-hour  calculations  are  accomplished  during  the  data  reduction 
and  analysis  phase  for  each  maintenance  scenario  specified. 

Skill  Level  Estimates 


Skill  level  estimates  should  be  made  in  conjunction  with  crew 
size  estimates  for  a  given  maintenance  scenario.  Have  technicians 
estimate  the  skill  levels  (3,  5,  7,  or  9)  for  the  member(s)  of  the 
crew(s)  they  estimated  would  be  required  to  perform  the  task.  The 
following  formats  (with  typical  estimates)  illustrate  how  the  skill 
level  estimates  may  be  made  to  correspond  to  the  crew  size  estimates. 


Maintenance  Scenario 

Component :  Transmitter  (WUC-ABAOO) 

Type  Maintenance:  On-Equipment _ 

How  Malfunction  Code:  No  Output 
Action  Taken  Code:  Troubleshoot 

Crew  Size  Percent  of  Time  Skill  Levels 

1 

2 

3 

4 


In  the  example,  the  estimator  indicates  that  when  one  person  can 
perform  the  task,  the  5-skill  level  is  necessary.  When  two  persons 
perform  the  task,  one  should  have  a  5-skill  level,  while  the  other 
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should  have  a  7-skill  level.  When  three  people  perform  the  task, 
skill  levels  5,  7,  and  3  are  required.  Since  four  people  are  not 
required  to  perform  the  task,  no  skill  level  estimates  are  necessary. 

Task  Difficulty  Estimates 

Task  difficulty  estimates  can  be  made  for  each  maintenance 
scenario.  Technicians  rate  the  difficulty  of  the  maintenance  action 
taken  for  the  proposed  equipment  on  a  task  difficulty  scale  100  mm 
long.  Verbal  anchors  of  "Very  Easy"  (0  mm),  "Average  Difficulty" 

(50  mm),  and  "Very  Difficult"  (100  mm)  are  added.  Technicians 
should  be  advised  to  consider  their  past  maintenance  experience  on 
similar  equipment  as  a  basis  for  rating  the  difficulty  of  the  main¬ 
tenance  tasks  for  the  proposed  equipment.  An  example  of  a  task 
difficulty  scale  is  as  follows. 


Maintenance  Scenario 

ComDonent:  Transmitter  (WUC-ABAOO) 

Tvoe  Maintenance :  On-Equipment 

How  Malfunction  Code:  No  Output 
Action  Taken  Code:  Troubleshoot 

Very 

Average 

Very 

Easy 

Difficulty 

Difficult 

Career  Field/AFSC  Estimates 


Career  field/AFSC  estimates  need  only  be  made  once,  since  they 
do  not  vary  by  maintenance  action.  It  is  recommended  that  these 
estimates  be  made  after  the  technicians  have  completed  the  estimates 
for  the  various  maintenance  scenarios.  It  is  helpful  to  identify 
for  the  estimators  the  group  of  current  AFSCs  from  which  technicians 
would  most  likely  be  chosen  to  maintain  the  new  system.  This  infor¬ 
mation  can  be  found  in  the  Airman  Classification  Regulation,  AFR  39- 
1.  A  series  of  questions  is  recommended  to  extract  as  much  informa¬ 
tion  from  the  estimators  as  possible.  The  questions  should  progress 
from  an  identification  of  the  appropriate  career  field  (the  first 
three  digits  of  an  AFSC)  to  identification  of  a  specific  AFSC.  Ask 
those  technicians  who  cannot  identify  an  appropriate  career  field  or 
AFSC  for  the  new  system  what  steps  would  be  required  to  obtain  the 
necessary  skills.  The  following  is  an  example  of  a  series  of 
career  field/AFSC  questions. 
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Career  Field/AFSC  Estimates 


In  your  opinion,  could  maintenance  on  this  radar  system  be 
performed  by  personnel  from  existing  Air  Force  career  fields? 
Yes  _ _  No  _ 

If  you  answered  yes,  which  career  field  do  you  think  would  be 
most  likely  to  perform  this  maintenance? 


Is  there  an  AFSC  in  existence  now  which  would  be  most  likely  to 
perform  the  maintenance  on  this  radar  system?  Yes  _  No  _ 

If  you  answered  yes,  please  identify  the  AFSC  _  X 

(AFSC) 

If  you  answered  no,  which  of  the  following  actions  do  you  think 
would  be  necessary  to  obtain  the  skills  for  this  system? 

_  Create  a  new  AFSC 

_  Combine  existing  AFSCs 

_  Create  a  new  shredout  for  an  existing  AFSC 


Training  Time  Estimates 

Training  time  estimates  can  be  collected  for  three  types  of 
training:  (a)  technical  training,  (b)  field  training  detachment 

(FTD)  training,  and  (c)  on-the-job  training.  The  estimates  should 
be  made  in  terms  of  the  number  of  weeks  required  for  each  type  of 
training.  Research  (Sauer  &  Askren,  1978)  has  shown  that  attempts 
to  have  technicians  provide  very  detailed  training  time  (in  hours) 
and  training  syllabus  content  estimates  during  the  early  stages  of 
design  were  not  productive.  Training  time  estimates  may  be  made  for 
two  types  of  individuals:  an  experienced  technician  transitioning 
from  a  similar  technology  to  the  new  system  or  a  new  airman,  a  basic 
training  graduate  ready  to  enter  technical  training  in  the  techno¬ 
logy  associated  with  the  proposed  system.  The  following  is  an 
example  of  the  format  used  to  collect  training  time  estimates. 
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Training  Time  Estimates 


Assume  that  maintenance  personnel  for  this  system  will  come 
from  both  experienced  radar  maintenance  technicians  and  Air 
Force  basic  training  graduates.  Further,  assume  that  the 
experienced  technicians  have  maintained  the  AN/TPS-44,  a  two- 
dimensional,  nontracking,  mobile,  ground-based  radar.  The 
AN/TPS-44  consists  of  an  antenna  assembly,  a  van  housing  the 
transmitter,  and  a  second  van  housing  the  PPI  display.  Esti¬ 
mate  the  training  time  in  weeks  required  to  accomplish  the 
following  types  of  training  for  both  experienced  technicians 
and  basic  training  graduates. 

Number  of  Weeks  of  Training 
_ Required  for: _ 

Basic 

Experienced  Training 

Technicians  Graduates 
Technical  Training  _  _ 

Field  Training  Detachment 

Training  _  _ 

On-the-Job  Training  _  _ 


Notes  on  the  Questionnaire 

It  is  recommended  that  verbal  directions  be  used  to  supplement 
the  written  instructions.  It  is  strongly  recommended  that  sample 
questions  be  included  with  the  instructions  to  demonstrate  how  the 
formats  are  to  be  used.  Technicians  may  never  have  encountered  some 
of  these  formats  and  an  example  may  be  the  most  effective  means  to 
resolve  any  confusion. 
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COLLECTING  EXPERT  ESTIMATES 


The  major  activities  involved  in  collecting  expert  estimates 
are  selecting  expert  estimators  and  conducting  the  actual  field  data 
collection  visits.  The  procedures  and  recommendations  for  these 
phases  of  the  method  are  presented  below. 

Select  Expert  Estimators 

Expert  estimators  should  be  selected  primarily  on  the  basis  of 
their  systems  experience.  Research  has  shown  that  it  is  desirable 
to  select  as  estimators  the  maintenance  technicians  with  experience 
on  equipment  or  systems  similar  to  the  proposed  equipment  or  system. 
Satisfactory  estimates  have  been  obtained  using  technician  esti¬ 
mators  from  the  career  field  (first  three  digits  of  the  AFSC) 
appearing  to  be  most  closely  associated  with  the  proposed  system. 

If  it  is  possible  to  identify  the  AFSC  (all  five  digits  of  AFSC) 
that  appears  to  be  most  qualified  to  maintain  the  proposed  system, 
use  estimators  from  that  AFSC.  Because  of  the  variety  of  new  sys¬ 
tems  under  development,  it  is  impossible  to  give  any  more  specific 
advice  on  the  ideal  equipment  or  systems  experience  necessary. 

Air  Force  Regulation  39-1,  Airman  Classification  Regulation, 
and  the  Air  Force  Manpower  and  Personnel  Center,  Randolph  AFB, 

Texas,  are  good  initial  sources  for  locating  technicians  who  could 
participate  as  estimators  and  for  identifying  the  Commands  to  which 
these  technicians  are  assigned.  Personnel  specialists  at  the 
various  Command  headquarters  can  provide  further  assistance  in 
locating  the  appropriate  technicians  within  their  Command. 

Once  the  initial  group  of  potential  estimators  has  been  iden¬ 
tified,  it  is  possible  to  select  specific  raters.  In  terms  of  skill 
level,  it  is  recommended  that  at  least  5- level  personnel  be  selected 
as  estimators.  Research  on  expert  estimators  has  not  included 
sufficient  numbers  of  3-level  personnel  to  assess  the  quality  of 
their  estimates.  Research  has  also  shown  that  the  amount  of  system 
experience  is  not  related  to  accuracy  of  estimates,  provided  the 
estimators  are  at  least  at  5-skill  level. 

The  minimum  recommended  number  of  expert  estimators  is  25. 
Previous  analyses  (Sauer  &  Askren,  1978)  of  estimates  from  groups  of 
5,  10,  20,  and  25  indicate  that  the  variability  in  estimates 
decreases  as  the  groups  increase  in  size.  There  is  less  chance, 
therefore,  of  obtaining  extremely  high  or  low  estimates  with  the 
larger  groups.  The  person- hour  cost  data  from  recent  research 
(Sauer,  Deem,  &  Askren,  1980)  show  only  a  relatively  small  addi¬ 
tional  cost  when  increasing  the  number  of  estimators  from  10  to  25. 
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Data  Collection  Visits 


After  the  appropriate  technicians  have  been  selected  and 
located,  it  is  necessary  to  prepare  for  the  data  collection  visits. 

A  certain  amount  of  lead  time  is  necessary  to  schedule  these  visits. 
At  least  6  weeks  advance  scheduling  is  recommended.  A  number  of 
variables  affect  the  actual  amount  of  lead  time  necessary  for  a 
given  situation.  Some  of  the  variables  are  security  clearance 
procedures,  command  visit  request  procedures,  type  of  personnel 
(military,  civil  service,  or  contractor)  making  the  visits,  and 
operations  or  deployment  schedules  of  the  units  or  squadrons  to  be 
visited. 

Once  the  visits  have  been  arranged,  attention  can  be  turned  to 
the  actual  data  collection  procedures.  The  data  may  be  collected  in 
group  sessions  ranging  from  2  to  15  technicians  per  session.  The 
length  of  each  session  will,  of  course,  vary  with  the  number  of 
estimates  to  be  made.  If  a  large  number  of  estimates  are  to  be 
made,  multiple  data  collection  sessions  should  be  scheduled.  In 
this  situation,  the  same  estimators  may  participate  in  multiple 
sessions.  An  alternative  would  be  to  divide  the  estimates  into 
separate  blocks,  with  each  block  assigned  to  a  different  but  equally 
qualified  group  of  estimators.  In  this  approach,  the  estimators 
would  only  be  required  for  a  single  session. 

Experience  with  the  expert  estimate  method  has  shown  that,  for 
planning  purposes,  one  should  expect  five  technicians  per  session 
and  each  data  collection  session  would  require  approximately  4  hours 
of  the  data  collector's  time.  These  planning  data  allow  for  varia¬ 
tions  in  the  unit's  work  schedule.  Depending  on  the  work  schedule, 
it  may  also  be  necessary  to  schedule  sessions  during  second  or  third 
shift  operations  to  obtain  the  required  number  of  estimators. 

The  data  collection  session  usually  begins  with  a  short 
briefing  on  the  purpose  of  the  data  collection.  This  often 
increases  the  motivation  of  technicians  to  complete  the  question¬ 
naires  as  carefully  and  accurately  as  possible.  Verbal  instructions 
should  supplement  the  written  instructions  for  the  questionnaire. 
Technicians  should  be  encouraged  to  first  skim  the  engineering 
description  package  and  then  refer  to  it  as  often  as  necessary 
during  the  session.  Point  out  that  although  the  questionnaire  is 
not  a  test,  individual  answers,  rather  than  a  group  consensus, 
produce  the  best  estimated  data.  Experience  thus  far  in  the 
research  has  shown  that  technicians  require  approximately  1  minute 
per  page  for  initial  reading  of  the  engineering  description  and 
about  1  minute  per  estimate.  Technician  estimators  in  the  research 
efforts  to  date  have  spent  from  1.5  to  2.5  hours  completing  the 
questionnaires. 
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DATA  REDUCTION  AND  ANALYSIS 


The  scope  of  the  expert  estimate  method  research  effort  has  not 
been  large  enough  to  require  data  processing  support  for  data  reduc¬ 
tion  and  analysis.  Data  reduction  has  only  involved  transferring 
data  from  the  questionnaires  to  a  consolidated  data  form.  These 
forms  simply  provide  a  convenient  method  to  handle  the  estimated 
data. 


The  analysis  of  the  estimated  data  is  fairly  straightforward. 
The  analysis  procedures  differ,  however,  for  the  various  types  of 
estimates.  The  product  of  the  analysis  is  an  estimate  of  a  human 
resource  data  item.  In  some  cases,  this  estimate  is  calculated 
directly  from  the  data.  In  other  cases,  a  correction  factor  has 
been  applied  to  the  basic  data  to  arrive  at  an  estimate.  The  fol¬ 
lowing  paragraphs  will  describe  the  analysis  procedures  for  the 
maintenance,  manpower,  and  training  uata  and  will  note  any  cor¬ 
rection  factors  which  may  be  applied  to  the  data. 

Maintenance  Task  Time  Estimates 


Maintenance  task  time  estimates  for  a  new  system  are  determined 
by  finding  the  mean  of  the  normal  task  time  estimates  made  by  the 
estimator  group.  If  five  technicians  produce  normal  task  time 
estimates  of  1.4,  1.7,  2.0,  1.9,  and  2.1  hours,  the  mean  task  time 
estimate  for  the  group  would  be  1.82  hours  [(1.4  +  1.7  +  2.0  +  1.9  + 
2. l)/5] .  The  minimum  and  maximum  time  estimates  serve  only  to  help 
the  estimator  make  the  normal  time  estimate.  Calculation  schemes 
using  all  three  task  time  estimates  were  tried  in  the  analysis  phase 
of  previous  studies  (Sauer  &  Askren,  1978;  Sauer,  Deem,  &  Askren, 
1980).  None  resulted  in  more  accurate  task  time  estimates. 

A  correction  factor  may  be  applied  to  the  maintenance  task  time 
estimates.  Based  on  previous  research  (Whalen  &  Askren,  1974), 
technicians  tend  to  underestimate  maintenance  task  time  by  approxi¬ 
mately  71  percent.  To  arrive  at  a  corrected  group  estimate  of 
maintenance  task  time,  divide  the  group  estimate  by  .71.  If  the 
group  mean  task  time  estimate  were  1.9  hours,  the  corrected  group 
mean  task  time  estimate  would  be  2.7  hours. 

Crew  Size  Estimates 


Analysis  of  the  crew  size  estimates  can  result  in  group  mean 
estimates  of  the  percentage  of  times  various  crew  sizes  would  be 
required  for  the  task  or  a  group  estimate  of  the  single  most  likely 
crew  size  required  for  the  task.  For  either  estimate,  calculate  the 
group  mean  percentage  for  each  crew  size.  These  percentages  will 
indicate  how  often  the  task  will  be  performed  by  one,  two,  three, 
four,  or  more  crewmembers.  The  crew  size  most  likely  to  perform  the 
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task  is  the  crew  size  with  the  largest  mean  percentage.  For  a  given 
maintenance  task,  five  technicians  estimated  the  crew  sizes  as 
follows: 


Crew 

Size 

Technician 

A 

Technician 

B 

Technician 

C 

Technician 

D 

Technician 

E 

Group 

Mean 

1 

30% 

15% 

40% 

50% 

10% 

29% 

2 

70% 

75% 

60% 

50% 

90% 

69% 

3 

0% 

5% 

0% 

0% 

0% 

1% 

4 

0% 

5% 

0% 

0% 

0% 

1% 

In  this  case,  a  crew  size  of  one  would  be  required  29  percent 
of  the  times  the  task  is  performed  and  a  crew  size  of  two  would  be 
required  69  percent  of  the  times.  Crews  of  three  or  four  would  be 
required  only  1  percent  of  the  time  the  task  was  performed.  The 
crew  size  most  likely  to  perform  this  task  is  two. 

The  research  has  focused  primarily  on  crew  sizes  of  one  and 
two.  Until  additional  data  can  be  obtained  for  crew  sizes  of  three 
and  four,  it  is  not  possible  to  recommend  any  correction  factors  for 
crew  size  estimates. 

Person-Hour  Values 


Individual  person-hour  values  are  calculated  by  multiplying  an 
individual  technician's  estimate  of  the  most  likely  crew  size  (indi¬ 
cated  by  the  largest  percentage)  by  the  individual's  estimate  of 
normal  task  time.  A  group  mean  person-hour  value  is  then  calculated 
from  the  individual  person-hour  values.  If  a  technician  estimates 
that  a  given  task  can  be  accomplished  in  2.1  hours  (normal  task 
time)  and  that  a  crew  size  of  two  is  most  likely  to  perform  the 
task,  the  person-hour  value  is  found  by  multiplying  2.1  (normal  task 
time)  by  2  (most  likely  crew  size)  which  yields  4.2  person-hours. 

If  person-hour  values  calculated  for  the  four  remaining  technicians 
in  the  group  were  3.9,  3.8,  4.7,  and  4.0,  the  group  mean  would  be 
4.12  person-hours  [(4.2  +  3.9  +  3.8  +  4.7  +  4.0)/5], 

The  research  has  shown  that  person-hour  values  ranged  from 
66  percent  to  74  percent  of  the  AFM  66-1  person-hours  used  to  vali¬ 
date  the  estimates.  In  other  words,  when  the  group  person-hour 
values  for  specific  on-equipment  maintenance  tasks  were  compared 
with  the  reported  AFM  66-1  person-hour  data  for  these  same  tasks, 
the  group  person-hour  values  were  less  than  the  actual  person-hours. 
To  bring  the  group  person-hour  values  closer  to  the  actual  person- 
hours,  a  correction  factor  may  be  applied  to  the  group  value.  Until 
additional  data  are  available,  the  correction  factor  for  person-hour 
estimates  is  .70.  This  value  is  midway  between  the  values  derived 
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from  the  Sauer  and  Askren  (1978)  study  (.66)  and  the  Sauer,  Deem, 
and  Askren  (1980)  study  (.74)  and  is  very  close  to  the  value  for 
task  times  (not  person-hours)  derived  from  the  Whalen  and  Askren 
(1974)  study  (.71).  To  arrive  at  a  corrected  person-hour  value, 
divide  the  group  person-hour  value  by  the  correction  factor  of  .70. 
This  yields  corrected  person-hours.  For  example,  if  the  group 
person-hour  value  for  a  given  task  was  1.76,  the  corrected  person- 
hours  would  be  found  by  dividing  1.76  by  .70.  The  corrected  person- 
hours  would  be  2.51. 

Skill  Level  Estimates 


The  analysis  of  skill  level  estimates  is  fairly  straightforward 
for  crew  sizes  of  one  and  two.  Those  skill  levels  reported  most 
often  by  the  majority  of  estimators  would  represent  the  predicted 
skill  level  under  the  particular  maintenance  scenario.  Although 
skill  level  estimate  data  for  crew  sizes  of  three  and  four  are 
limited,  it  seems  reasonable  to  obtain  skill  level  estimates  for 
these  crew  sizes  in  the  same  manner.  For  example,  suppose  five 
technicians  estimated  skill  levels  for  a  given  task  as  follows: 

Skill  Level  Estimates 


Crew 

Size 

Technician 

A 

Technician 

B 

Technician 

C 

Technician 

D 

Technician 

E 

1 

5 

5 

5 

5 

7 

2 

5,  7 

5,  7 

5, 

3 

5,  5 

5,  7 

3 

3,  5,  7 

3,  3,  7 

5,  5, 

3 

5, 

3,  7 

3,  5,  7 

4 

- 

- 

5, 

5,  3, 

3 

- 

- 

The  estimated 

skill  level 

for 

a  crew 

size 

of 

one  would 

be  5 

(4/5,  or  80  percent,  of  the  responses)  and  for  a  crew  size  of  two, 
the  estimated  skill  levels  would  be  a  5  and  a  7  (3/5,  or  60  percent, 
of  the  responses).  For  a  crew  size  of  three,  the  estimated  skill 
levels  would  be  3,  5,  and  7  (3/5,  or  60  percent,  of  the  responses). 
Although  it  is  doubtful  that  a  crew  of  four  would  be  required  for 
the  task,  the  skill  levels  estimated  for  this  size  would  be  two  5- 
level  and  two  3-level  technicians.  No  correction  factors  are 
involved  with  skill  level  estimates. 

Task  Difficulty  Estimates 

Task  difficulty  estimates  are  calculated  by  converting  the 
marks  on  the  100-mm  difficulty  scale  to  scores.  If  five  technicians 
estimated  task  difficulty  as  75,  72,  55,  48,  and  64,  the  group  mean 


task  difficulty  estimate  could  be  62.8,  indicating  that  the  task  is 
somewhat  more  difficult  than  average  tasks.  After  calculating  the 
group  mean  task  difficulty  score  for  a  particular  task,  a  correction 
should  be  applied.  Previous  research  has  shown  that  technicians 
tend  to  underestimate  the  degree  of  difficulty  of  maintenance  tasks 
on  the  proposed  system  by  from  74  percent  (Sauer  &  Askren,  1978)  to 
88  percent  (Sauer,  Deem,  &  Askren,  1980).  That  is,  technicians 
rated  the  maintenance  tasks  on  the  proposed  system  as  being  less 
difficult  than  the  criteria  task  difficulty  ratings  for  the  same 
tasks.  To  obtain  a  corrected  task  difficulty  estimate,  the  mean 
task  difficulty  estimate  must  be  divided  by  a  correction  factor  of 
.81,  a  value  midway  between  the  percentages  reported  in  the  above 
studies.  For  a  task  difficulty  estimate  of  54,  the  corrected  task 
difficulty  estimate  would  be  found  by  dividing  54  by  .81,  which 
would  result  in  a  corrected  estimate  of  66.7. 

Career  Field  AFSC  Estimates 


The  career  field  and  AFSC  estimates  for  a  new  system  are  simply 
the  career  field  and  AFSC  identified  by  the  majority  of  the  techni¬ 
cians  as  appropriate  for  the  proposed  system.  If  a  career  field  or 
AFSC  is  not  clearly  identified  by  the  technicians,  the  other 
responses  to  the  career  field  and  AFSC  questions  should  indicate 
whether  a  new  career  field  (first  three  digits  of  AFSC)  or  new 
specialty  (all  five  digits  of  AFSC)  should  be  created  or  whether 
other  modifications  to  existing  AFSCs  may  be  necessary.  No  cor¬ 
rection  factors  are  appropriate  for  career  field/AFSC  estimates. 

Training  Time  Estimates 

The  first  step  in  determining  training  time  estimates  is  to 
calculate  the  group  mean  of  the  individual  training  time  estimates. 
The  mean  training  time  estimates  may  be  collected  for  three  types  of 
training:  (a)  technical  training,  (b)  field  training  detachment 
training,  and  (c)  on-the-job  training;  and  two  types  of  technicians: 
(a)  the  experienced  maintenance  technician  transitioning  to  the  new 
system,  and  (b)  the  basic  training  graduate  ready  to  begin  technical 
training  for  the  new  system.  Suppose  10  technicians  estimate  the 
number  of  weeks  of  technical  training  required  for  a  basic  training 
graduate.  Their  estimates  of  10,  6,  8,  10,  14,  16,  12,  11,  10,  and 
8  weeks  result  in  a  mean  estimate  of  10.5  weeks  for  a  basic  training 
graduate.  Other  training  time  estimates  are  calculated  in  a  similar 
manner. 

Correction  factors  (Sauer,  Deem,  &  Askren,  1980)  may  be  applied 
to  all  these  estimates.  The  correction  factor  for  training  time 
estimates  is  handled  in  the  same  way  as  correction  factors  for 
maintenance  task  times  and  person-hours.  The  estimated  training 
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times  are  divided  by  the  accuracy  scores  to  yield  a  corrected 
training  time  estimate.  Table  1  contains  the  accuracy  scores  or 
correction  factors  for  each  of  the  three  types  of  training  and  two 
types  of  technicians. 

TABLE  1.  TRAINING  TIME  ESTIMATE  CORRECTION 

FACTORS  FOR  THREE  TYPES  OF  TRAINING  AND 
TWO  TYPES  OF  TECHNICIANS 


Types  of 

Types  of  Training 

Technicians 

Technical 

Field  Training  Detachment 

On-the-Job 

Experienced 

Technician 

.63 

1.41 

.89 

Basic  Training 
Graduate 

1.13 

2.74 

.83 

For  example,  if  the  group  of  technicians  estimated  that,  for  an 
experienced  technician,  10  weeks  of  on-the-job  training  would  be 
required,  the  correction  factor  would  be  .89.  The  corrected 
training  time  estimate  would  be  10  divided  by  .89  or  11.24  weeks  of 
on-the-job  training. 


20 


REFERENCES 


Air  Force  Manual  66-1.  Maintenance  management  policy . 

Washington,  D.C.:  Department  of  the  Air  Force,  1  November 
1975. 

Air  Force  Regulation  39-1.  Airman  Classification  Regulation. 

Washington,  D.C.:  Department  of  the  Air  Force,  1  June  1977. 

Haines,  D.  B. ,  &  Gael,  S.  Estimating  manning  requirements  for 

advanced  systems :  _a  survey  of  the  defense  industry.  AMRL-TDR 
63-110.  Wright-Patterson  AFB,  OH.  Behavioral  Sciences  Labora 
tory,  6570th  Aerospace  Medical  Research  Laboratories,  November 
1963. 


Sauer,  D.  W. ,  &  Askren,  W.  B.  Validation  of  an  expert  estimate 

technique  for  predicting  manpower ,  maintenance ,  and  training 
requirements  for  proposed  Air  Force  systems .  AFHRL-TR-78-19, 

AD-A058  851.  Wright-Patterson  AFB,  OH,  Advanced  Systems 
Division,  Air  Force  Human  Resources  Laboratory,  May  1978. 

Sauer,  D.  W. ,  Deem,  R.  N.,  &  Askren,  W.  B.  Expert  estimate  method 
of  generating  maintenance  and  manpower  data  for  proposed  Air 
Force  Systems :  Evaluation.  AFHRL-TR-79-79 .  Wright-Patterson 
AFB,  OH:  Advanced  Systems  Division,  Air  Force  Human  Resources 
Laboratory,  February  1980. 

Whalen,  G. ,  &  Askren,  W.  B.  Impact  of  design  trade  studies  on 

system  human  resources .  AFHRL-TR-74-89 ,  AD-A009  639.  Wright- 
Patterson  AFB,  Ohio.  Advanced  Systems  Division,  Air  Force 
Human  Resources  Laboratory,  December  1974. 


GLOSSARY 


AFTO  Form  349 — The  title  of  this  form  is  the  "Maintenance  Data 

Collection  Record."  It  is  used  to  record  maintenance  actions 
on  various  types  of  equipment.  The  data  which  may  be  recorded 
include:  job  identification  data,  component  identification 

data,  maintenance  task  time,  crew  size,  type  of  maintenance, 
malfunctions,  discrepancies,  and  specific  maintenance  actions 
taken. 

Air  Force  Regulation  39-1 — The  Airman  Classification  Regulation  defines 
and  describes  all  Air  Force  occupational  specialties  and  career 
fields  and  their  training  and  skill  requirements. 

Air  Force  Manual  66-1 — This  manual,  "Maintenance  management  policy," 
establishes  the  maintenance  management  system  applicable  to  all 
Air  Force  activities  engaged  in  the  maintenance  of  aircraft, 
missiles,  munitions,  aerospace  ground  equipment,  avionics, 
training  equipment,  and  communications-electronics-meteoro- 
logical  equipment. 

Air  Force  Specialty  Code — The  Air  Force  Specialty  Code  (AFSC)  is  a 
five-digit  number  assigned  to  Air  Force  enlisted  personnel 
which  defines  the  general  career  field  and  the  specialty  within 
the  career  field  in  which  the  individual  is  qualified  and  the 
skill  level  the  individual  has  attained. 

Career  Field — Career  Field  represents  a  general  area  of  expertise  or 
technology  in  which  an  individual  is  qualified.  The  first 
three  digits  of  the  Air  Force  Specialty  Code  (AFSC)  specify  the 
individual's  career  field.  For  example,  the  career  field  of 
avionic  weapon  delivery  systems  (general  area)  will  include 
three  specialty  areas:  bomb-navigation  system,  defensive  fire 
control  systems,  and  weapon  control  systems. 

Field  Training  Detachment  Training — Field  Training  Detachments 

provide  job-oriented  maintenance  training  on  assigned  weapon 
systems,  support  systems,  and  selective  equipment.  The  FTDs 
require  host  support  in  order  to  provide  responsive  and  quality 
training. 

Off-Equipment  Maintenance — Maintenance  activities  which  cannot  be 
accomplished  on  the  system.  Maintenance  must  be  accomplished 
in  the  maintenance  shop. 

On-Equipment  Maintenance — Maintenance  actions  which  may  be  performed 
on  the  system.  No  shop  maintenance  is  required. 
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On-The-Job  Training — Training  which  is  accomplished  in  the  course  of 
normal  operations  or  maintenance  activities.  The  trainee 
learns  by  observing  and/or  actually  performing  real  operations 
or  maintenance  tasks. 

Skill  Level — Skill  level  represents  the  level  of  qualification  and 
degree  of  expertise  achieved  within  the  technician's  career 
field  and  specialty  code.  The  five  skill  level  codes  are  1,  3, 
5,  7,  and  9,  with  skill  level  1  equivalent  to  a  helper,  skill 
level  3  equivalent  to  an  apprentice,  skill  level  5  equivalent 
to  a  specialist,  skill  level  7  equivalent  to  a  technician,  and 
skill  level  9  equivalent  to  a  superintendent.  This  code  appears 
as  the  fourth  digit  of  an  individual's  Air  Force  Specialty  Code 
(AFSC) . 

Technical  Training — Formal  classroom  training  conducted  away  from 
the  job  site  at  designated  Technical  Training  Centers. 

Work  Unit  Code — The  work  unit  code  is  a  five-character  code  used  to 
identify  systems,  subsystems,  and  components  for  which  mainte¬ 
nance  is  required  or  on  which  maintenance  was  accomplished. 
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